Recognizing Facial Expressions Automatically from Video

Facial expressions, resulting from movements of the facial muscles, are the face changes in response to a person’s internal emotional states, intentions, or social communications. There is a considerable history associated with the study on facial expressions. Darwin (1872) was the first to describe in details the specific facial expressions associated with emotions in animals and humans, who argued that all mammals show emotions reliably in their faces. Since that, facial expression analysis has been a area of great research interest for behavioral scientists (Ekman, Friesen, and Hager, 2002). Psychological studies (Mehrabian, 1968; Ambady and Rosenthal, 1992) suggest that facial expressions, as the main mode for non-verbal communication, play a vital role in human face-to-face communication. For illustration, we show some examples of facial expressions in Fig. 1. Computer recognition of facial expressions has many important applications in intelligent human-computer interaction, computer animation, surveillance and security, medical diagnosis, law enforcement, and awareness systems (Shan, 2007). Therefore, it has been an active research topic in multiple disciplines such as psychology, cognitive science, human-computer interaction, and pattern recognition. Meanwhile, as a promising unobtrusive solution, automatic facial expression analysis from video or images has received much attention in last two decades (Pantic and Rothkrantz, 2000a; Fasel and Luettin, 2003; Tian, Kanade, and Cohn, 2005; Pantic and Bartlett, 2007). This chapter introduces recent advances in computer recognition of facial expressions. Firstly, we describe the problem space, which includes multiple dimensions: level of description, static versus dynamic expression, facial feature extraction and